Improving on reality: can simulation facilitate practice change?

ANESTHESIOLOGISTS pioneered simulation, as a technology and an educational methodology, in the late 1980s. Yet, its adoption has been limited largely to residency training programs and medical schools. A number of institutions and researchers have reported the benefits of simulation for teaching medical students, anesthesiology residents, and trainees in other disciplines. The benefit of simulation for teaching procedural skills has been shown to transfer to the bedside. Although the benefit of simulation for training cognitive and behavioral skills has been shown to improve performance in the simulation laboratory, similar improvements are harder to measure in a clinical setting. Appropriately, using the simulated setting for measuring outcomes raises concerns over whether the performance improvement is a result of “teaching to the test,” and whether performance gains are transferable to patients. In this issue of ANESTHESIOLOGY, the link between simulation and clinical care has finally been made. Bruppacher et al. show that 2 h of simulation training, when compared with a similar duration of training via an interactive seminar, results in improved patient care during weaning from cardiopulmonary bypass 2 and 5 weeks after the training. The primary and secondary outcome measures were “nontechnical” and “technical” skills, respectively. The nontechnical skills, defined as behavioral skills such as teamwork and decision making, were measured by the Anesthesiologists Non-Technical Skills (ANTS) rating scale, a previously described and widely used rating. The technical skills, or discrete tasks, were measured using a checklist developed for this study using the Delphi method. It is unlikely that the two scoring techniques are entirely independent, yet the two scales viewed in aggregate probably represent global performance. Regardless, the scores of the simulation group were 20% higher than that of the seminar group at 2 and 5 weeks after the training. A common measure of training effectiveness is the effect size of an intervention. An effect size of 1 standardized mean is considered large, whereas 0.6–0.8 is moderate and 0.2– 0.4 is slight. Although Bruppacher et al. reported standard error rather than standard deviation, the effect size for the two interventions can be easily ascertained and is nearly 2 for the seminar group and twice that for the simulation group. This illustrates that although the seminar had a sizable impact, the simulation group’s gains are even more impressive by comparison. Are these improvements so large that they are unreasonable? A scoring bias may have lowered pretest scores. Raters were not aware of the group assignment of the subjects but were aware that none of the subjects had received training and, therefore, may have been reluctant to give high scores. Similarly, all scores could have been inflated after the intervention; however, this would not explain the larger improvements in the simulation group compared with the seminar group, which occurred in each of the four categories in the ANTS scale (as shown in table 3). The checklist, less susceptible to subjective interpretation, supports the main findings. Interestingly, the interrater reliability of the checklist was lower than that of the ANTS scale. This may be due to the difficulty encountered when attempting to observe a number of discrete actions without the benefit of videotape or may have been due to ambiguities in the descriptions of the items. Experience with the checklist could have minimized these difficulties. In favor of the plausibility of the large magnitude of the effect of training is the nature of the setting chosen by the authors. Weaning from cardiopulmonary bypass occurs in a complex clinical environment from both a technological and physiologic standpoint, and it is certainly conceivable that large improvements in patient management would occur as a result of training novices. Are the outcomes of the study sufficiently convincing? Is a better ANTS score something patients, or healthcare providers, would consider compelling? Perhaps not at first glance, but after reading the descriptors for the anchors in the ANTS categories, these attributes are certainly ones that every patient seeks during their care. Avoiding rescue interventions by an attending cardiac anesthesiologist is not identical to improved survival, but given the obvious gravity of the clinical setting, it is easy to understand why process matters in creating optimal outcomes. Furthermore, no 2-h training